Automating Data Quality Monitoring by Jeremy Stanley

Author:Jeremy Stanley
Language: eng
Format: epub
Publisher: O'Reilly Media
Published: 2024-01-09T00:00:00+00:00

In a dataset like this, an anomaly might occur at the product level (all Greek Yogurt is missing), which could be caught by features derived from the Aisle column (Yogurt is anomalous), Product column (Greek Yogurt), or Brand column (a collection of specific brands are anomalous). Department is probably too highly aggregated to be sensitive to an anomaly for Greek yogurt, and the Item column is too fragmented to be easily used to detect the anomaly.

Given that our algorithm has produced the SHAP-based anomaly scores for each individual record, we can apply clustering algorithms to those anomaly scores to detect that the anomalies in these columns are all happening on the same set of rows.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.